There are several ways to rename a custom group or a smart group; choose the one that is the most convenient for you.
To rename a group:
In the left panel of the Library window, click on the name of the group.
Select the Rename Group command in one of these ways:
Click on the group name a second time to directly edit the name.
From the Groups menu, select Rename Group.
Control+click Right-click with your mouse on the group name to display a menu, and select Rename Group.
You cannot rename the automatically created groups: All References, Duplicate References, Imported References, Search Results, Unfiled, Trash and the Find Full Text groups.